Assessing the performance of Internet Protocol (IP) based and other networks has long been a concern, not only for Internet Service Providers (ISPs) but also for users, standards bodies, network equipment vendors, regulators and others.
In relation to network and communications technology, the terms “Quality of Service” (QoS) and “Quality of Experience” (QoE) are often used, the former relating generally to an objective (or at least substantially objective) measure of performance in respect of the network or communications service provided to customers, and the latter relating generally to a subjective (or at least substantially subjective) measure indicative of how satisfied users are likely to be with the services provided to them.
“Quality of Service” can be defined and measured in various ways, taking account of various different criteria (including network characteristics such as loss, delay, jitter and throughput), but once defined, it can (at least theoretically) be measured, because such network characteristics can be measured, and values in respect thereof can then be combined (if required) according to a predetermined function in order to obtain one or more “QoS measures”, “QoS values” or other such “QoS indications”. “Quality of Experience”, while generally being dependent on the same or similar criteria, and therefore generally being dependent at least in part on the “Quality of Service” being achieved in respect of the service or services provided to an individual customer (and therefore generally correlating to some extent with the “Quality of Service” being achieved for that customer), is essentially a subjective measure, so cannot generally be measured directly, other than by obtaining feedback directly from customers as to their satisfaction levels with the services concerned (which themselves will generally be subjective—they may be affected by the mood of the customer concerned, or whether the customer has previous experience of other comparable services being better or worse). “Quality of Experience” can however be estimated or assessed in various ways, generally taking account of one or more types of performance criteria such as those generally used to obtain “Quality of Service” measures, then estimating the effect(s) of these on the experience of a service, as it will be perceived by the customer. Thus, while “Quality of Experience” is generally a subjective measure, it can—once suitably defined—also be estimated, assessed or evaluated in an objective manner, based (at least partly) on measurable network performance data.
Methods exist for testing generic services and/or making network performance measurements in respect of various different characteristics. In relation to end-to-end testing of broadband services, two methods now predominate: (i) test websites; and (ii) probe-based tests.
With test websites, a browser application or “app” is used to conduct tests between a user's device and a remote test-server, e.g. www.speedtest.net by Ookla. A series of packets are sent between the user's device and a remote test-server allowing network characteristics such as Round Trip Time (RTT), throughput, jitter and loss to be calculated.
With probe-based tests, companies such as SamKnows (www.samknows.com) use a panel of users who are provided with a probe device for use in their homes/offices/local networks. These probe devices then perform tests involving exchanges with a number of test-servers to measure network characteristics. The probe devices may also test the behaviour of typical internet applications such as Voice over IP (VoIP) communication and web-browsing. Web-browsing may be tested by loading a few popular websites and recording page-load times, for example. The probe devices provided to the panel of users are normally paid for by the organisation wanting to do the testing, which is generally a network owner or a regulator.
An article entitled: “Measuring and Understanding Broadband: Speed, Quality and Application”, dated 8 Jun. 2010 from “Ookla” (available online at: http://www.ookla.com/docs/UnderstandingBroadbandMeasurement.pdf) discusses Ookla's recommended techniques. These are for web-based network performance measurements, and the article explains the authors' reasoning for measuring “ . . . the maximum sustainable throughput of a connection at a given location to and from a server of their choosing”.
A document entitled: “SAMKNOWS TEST METHODOLOGY—Methodology and technical information relating to the SamKnows testing platform” dated September 2014 from “SamKnows” having a Document Ref: SQ301-003-EN (available online at: https://www.samknows.com/broadband/uploads/methodology/SQ301-003-EN-Test-Suite-Whitepaper.pdf) discusses various performance tests that can be made in respect of broadband connections. This document discusses generic probe-based measurement of network performance characteristics, including some application specific testing.
Various prior art documents indicate that they aim to measure (or at least estimate) QoE rather than simply QoS or network performance, but these issues are often confused.
European patent application EP2632100 (for example) relates to systems and methods for traffic management, and discloses a traffic management system having a “ . . . quality of experience (QoE) measurement module configured to monitor a QoE metric in real-time”.
United States application US 2013/0262659 relates to measuring web-browsing quality of experience in real-time at an intermediate network node, based on the time taken for web-pages to be downloaded.
European patent application EP2784984 relates to techniques for monitoring network performance in respect of a digital communications network comprising a user-network having at least one user-device therein that is able to submit requests for data to and receive data from one or more remote servers via an intermediate control module, and an access network via which data may be delivered to the user-network via the control module, and in particular discloses techniques that allow a network service provider providing network services via an access network to its customers' home networks to have visibility on the performance of its customers' home networks and/or on the performance of one or more networked devices within those customers' home networks.
Current methods of QoE assessment/measurement in respect of broadband lines generally only measure the general performance of lines, however, typically measuring bandwidth and connectivity to a few dedicated test-servers. Some application-specific tests attempt to measure QoE for applications such as generic VoIP (to a dedicated test-server again) or for a particular service (e.g. a particular website or other web-service such as YouTube).
Assessing the Quality of Experience that customers achieve (or more correctly, perceive) in relation to networked services provided over their broadband lines or other such network connections is of great importance to network providers or ISPs, because how customers rate the service they receive from their internet-connected computing and telephony devices depends to a large extent on the QoE they perceive for internet-delivered services (as opposed to the (objective) Quality of Service (QoS) that is actually being achieved).
While methods exist for testing generic services, as indicated above, it will be noted that such methods are not directed to testing those services that individual customers actually use. Customers generally judge the performance of their ISP on the quality of the applications and services they are actually using or (frequently or generally) actually use, however. They may, for example, be communicating (or may frequently communicate) with relatives in another country or other countries using a particular VoIP service, accessing websites that are not on a generic, pre-determined test-schedule, using different video services, etc.
Moreover, types of network characteristic that can be taken into account in obtaining measurements of QoS (e.g. loss, delay, jitter, throughput, etc.) can affect different network-reliant applications in different ways and to different extents, so changes in relation to network characteristics that would generally have the same or a similar effect on respective QoS measures for different customers may well have very different effects on respective QoE measures for those different customers (or for the same customer at different times). This means that while the QoE perceived will generally still show some level of correlation with the QoS that is being achieved, the QoE at a particular time may be particularly affected by changes in respect of network characteristics that do not have such a significant effect on the QoS, or vice versa, thereby weakening or removing any such correlation. The extent to which any such correlation will be affected may depend on the extent to which the network characteristics concerned affect the network-reliant applications that are actually being used by the respective customers (or by the same customer at different times).
Referring briefly to other prior disclosures, International application WO2014/166523 (“Nokia”) relates to methods and apparatus which aim to provide insight into the user experience of web-based applications. One such method involves collecting and measuring application level Key Performance Indicators (KPIs), detecting user actions by monitoring network side user traffic in a network, correlating the user actions with the KPIs in order to evaluate and quantify a quality of experience (QoE) of the user, and correlating poor QoE with network side KPIs to determine an underlying root cause of the poor QoE.
A paper entitled “Building a Standard Measurement Platform” by Bagnulo et al (IEEE Communications Magazine, vol. 52, no. 5, May 2014) discusses how network management is achieved via different solutions for different technologies and parts of the end-to-end network, and how gaining an overall view and predicting the impact on a service user is difficult. It discusses how a number of proprietary platforms have emerged to conduct end-to-end testing from user premises, and how these are limited in scale, interoperability, and the ability to compare like-for-like results. Taking the SamKnows platform as a use case, an evolution is proposed from its current proprietary protocols to standardized protocols and tests.